notebook.community

Edit and run 





In [1]:



    


import tensorflow as tf
from tensorflow import data
import shutil
import math
from datetime import datetime
from tensorflow.python.feature_column import feature_column

from tensorflow.contrib.learn import learn_runner
from tensorflow.contrib.learn import make_export_strategy

print(tf.__version__)



    


















    






/Users/khalidsalama/anaconda/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
  return f(*args, **kwds)










    






1.4.0

Steps to use the TF Experiment APIs

  1. Define dataset metadata
  2. Define data input function to read the data from csv files + feature processing
  3. Create TF feature columns based on metadata + extended feature columns
  4. Define an estimator (DNNRegressor) creation function with the required feature columns & parameters
  5. Run an Experiment with learn_runner to train, evaluate, and export the model
  6. Evaluate the model using test data
  7. Perform predictions & serving the exported model (via json function)




In [2]:



    


MODEL_NAME = 'reg-model-03'

TRAIN_DATA_FILES_PATTERN = 'data/train-*.csv'
VALID_DATA_FILES_PATTERN = 'data/valid-*.csv'
TEST_DATA_FILES_PATTERN = 'data/test-*.csv'

RESUME_TRAINING = False
PROCESS_FEATURES = True
EXTEND_FEATURE_COLUMNS = True
MULTI_THREADING = True

1. Define Dataset Metadata

  • CSV file header and defaults
  • Numeric and categorical feature names
  • Target feature name
  • Unused columns




In [3]:



    


HEADER = ['key','x','y','alpha','beta','target']
HEADER_DEFAULTS = [[0], [0.0], [0.0], ['NA'], ['NA'], [0.0]]

NUMERIC_FEATURE_NAMES = ['x', 'y']  

CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY = {'alpha':['ax01', 'ax02'], 'beta':['bx01', 'bx02']}
CATEGORICAL_FEATURE_NAMES = list(CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.keys())

FEATURE_NAMES = NUMERIC_FEATURE_NAMES + CATEGORICAL_FEATURE_NAMES

TARGET_NAME = 'target'

UNUSED_FEATURE_NAMES = list(set(HEADER) - set(FEATURE_NAMES) - {TARGET_NAME})

print("Header: {}".format(HEADER))
print("Numeric Features: {}".format(NUMERIC_FEATURE_NAMES))
print("Categorical Features: {}".format(CATEGORICAL_FEATURE_NAMES))
print("Target: {}".format(TARGET_NAME))
print("Unused Features: {}".format(UNUSED_FEATURE_NAMES))



    


















    






Header: ['key', 'x', 'y', 'alpha', 'beta', 'target']
Numeric Features: ['x', 'y']
Categorical Features: ['alpha', 'beta']
Target: target
Unused Features: ['key']

2. Define Data Input Function

  • Input csv files name pattern
  • Use TF Dataset APIs to read and process the data
  • Parse CSV lines to feature tensors
  • Apply feature processing
  • Return (features, target) tensors

a. parsing and preprocessing logic





In [4]:



    


def parse_csv_row(csv_row):
    
    columns = tf.decode_csv(csv_row, record_defaults=HEADER_DEFAULTS)
    features = dict(zip(HEADER, columns))
    
    for column in UNUSED_FEATURE_NAMES:
        features.pop(column)
    
    target = features.pop(TARGET_NAME)

    return features, target

def process_features(features):

    features["x_2"] = tf.square(features['x'])
    features["y_2"] = tf.square(features['y'])
    features["xy"] = tf.multiply(features['x'], features['y']) # features['x'] * features['y']
    features['dist_xy'] =  tf.sqrt(tf.squared_difference(features['x'],features['y']))
    
    return features

b. data pipeline input function





In [5]:



    


def csv_input_fn(files_name_pattern, mode=tf.estimator.ModeKeys.EVAL, 
                 skip_header_lines=0, 
                 num_epochs=None, 
                 batch_size=200):
    
    shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False
    
    print("")
    print("* data input_fn:")
    print("================")
    print("Input file(s): {}".format(files_name_pattern))
    print("Batch size: {}".format(batch_size))
    print("Epoch Count: {}".format(num_epochs))
    print("Mode: {}".format(mode))
    print("Shuffle: {}".format(shuffle))
    print("================")
    print("")
    
    file_names = tf.matching_files(files_name_pattern)

    dataset = data.TextLineDataset(filenames=file_names)
    dataset = dataset.skip(skip_header_lines)
    
    if shuffle:
        dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)

    #useful for distributed training when training on 1 data file, so it can be shareded
    #dataset = dataset.shard(num_workers, worker_index)
    
    dataset = dataset.batch(batch_size)
    dataset = dataset.map(lambda csv_row: parse_csv_row(csv_row))
    
    if PROCESS_FEATURES:
        dataset = dataset.map(lambda features, target: (process_features(features), target))
    
    #dataset = dataset.batch(batch_size) #??? very long time
    dataset = dataset.repeat(num_epochs)
    iterator = dataset.make_one_shot_iterator()
    
    features, target = iterator.get_next()
    return features, target



    











In [6]:



    


features, target = csv_input_fn(files_name_pattern="")
print("Feature read from CSV: {}".format(list(features.keys())))
print("Target read from CSV: {}".format(target))



    


















    






* data input_fn:
================
Input file(s): 
Batch size: 200
Epoch Count: None
Mode: eval
Shuffle: False
================

Feature read from CSV: ['x', 'y', 'alpha', 'beta', 'x_2', 'y_2', 'xy', 'dist_xy']
Target read from CSV: Tensor("IteratorGetNext:8", shape=(?,), dtype=float32)

3. Define Feature Columns

The input numeric columns are assumed to be normalized (or have the same scale). Otherwise, a normlizer_fn, along with the normlisation params (mean, stdv or min, max) should be passed to tf.feature_column.numeric_column() constructor.





In [7]:



    


def extend_feature_columns(feature_columns):
    
    # crossing, bucketizing, and embedding can be applied here
    
    feature_columns['alpha_X_beta'] = tf.feature_column.crossed_column(
        [feature_columns['alpha'], feature_columns['beta']], 4)
    
    return feature_columns

def get_feature_columns():
    
    CONSTRUCTED_NUMERIC_FEATURES_NAMES = ['x_2', 'y_2', 'xy', 'dist_xy']
    all_numeric_feature_names = NUMERIC_FEATURE_NAMES.copy() 
    
    if PROCESS_FEATURES:
        all_numeric_feature_names += CONSTRUCTED_NUMERIC_FEATURES_NAMES

    numeric_columns = {feature_name: tf.feature_column.numeric_column(feature_name)
                       for feature_name in all_numeric_feature_names}

    categorical_column_with_vocabulary = \
        {item[0]: tf.feature_column.categorical_column_with_vocabulary_list(item[0], item[1])
         for item in CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.items()}
        
    feature_columns = {}

    if numeric_columns is not None:
        feature_columns.update(numeric_columns)

    if categorical_column_with_vocabulary is not None:
        feature_columns.update(categorical_column_with_vocabulary)
        
    if EXTEND_FEATURE_COLUMNS:
        feature_columns = extend_feature_columns(feature_columns)
        
    return feature_columns

feature_columns = get_feature_columns()
print("Feature Columns: {}".format(feature_columns))



    


















    






Feature Columns: {'x': _NumericColumn(key='x', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'y': _NumericColumn(key='y', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'x_2': _NumericColumn(key='x_2', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'y_2': _NumericColumn(key='y_2', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'xy': _NumericColumn(key='xy', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'dist_xy': _NumericColumn(key='dist_xy', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'alpha': _VocabularyListCategoricalColumn(key='alpha', vocabulary_list=('ax01', 'ax02'), dtype=tf.string, default_value=-1, num_oov_buckets=0), 'beta': _VocabularyListCategoricalColumn(key='beta', vocabulary_list=('bx01', 'bx02'), dtype=tf.string, default_value=-1, num_oov_buckets=0), 'alpha_X_beta': _CrossedColumn(keys=(_VocabularyListCategoricalColumn(key='alpha', vocabulary_list=('ax01', 'ax02'), dtype=tf.string, default_value=-1, num_oov_buckets=0), _VocabularyListCategoricalColumn(key='beta', vocabulary_list=('bx01', 'bx02'), dtype=tf.string, default_value=-1, num_oov_buckets=0)), hash_bucket_size=4, hash_key=None)}

4. Define an Estimator Creation Function

  • Get dense (numeric) columns from the feature columns
  • Convert categorical columns to indicator columns
  • Create Instantiate a DNNRegressor estimator given dense + indicator feature columns + params




In [8]:



    


def create_estimator(run_config, hparams):
    
    feature_columns = list(get_feature_columns().values())
    
    dense_columns = list(
        filter(lambda column: isinstance(column, feature_column._NumericColumn),
               feature_columns
        )
    )

    categorical_columns = list(
        filter(lambda column: isinstance(column, feature_column._VocabularyListCategoricalColumn) |
                              isinstance(column, feature_column._BucketizedColumn),
                   feature_columns)
    )

    indicator_columns = list(
            map(lambda column: tf.feature_column.indicator_column(column),
                categorical_columns)
    )
    
    
    estimator = tf.estimator.DNNRegressor(
        
        feature_columns= dense_columns + indicator_columns ,
        hidden_units= hparams.hidden_units,
        
        optimizer= tf.train.AdamOptimizer(),
        activation_fn= tf.nn.elu,
        dropout= hparams.dropout_prob,s
        config= run_config
    )

    print("")
    print("Estimator Type: {}".format(type(estimator)))
    print("")
    
    return estimator

5. Run Experiment

a. Define Experiment Function





In [9]:



    


def generate_experiment_fn(**experiment_args):

    def _experiment_fn(run_config, hparams):

        train_input_fn = lambda: csv_input_fn(
            files_name_pattern=TRAIN_DATA_FILES_PATTERN,
            mode = tf.estimator.ModeKeys.TRAIN,
            num_epochs=hparams.num_epochs,
            batch_size=hparams.batch_size
        )

        eval_input_fn = lambda: csv_input_fn(
            files_name_pattern=VALID_DATA_FILES_PATTERN,
            mode=tf.estimator.ModeKeys.EVAL,
            num_epochs=1,
            batch_size=hparams.batch_size
        )

        estimator = create_estimator(run_config, hparams)

        return tf.contrib.learn.Experiment(
            estimator,
            train_input_fn=train_input_fn,
            eval_input_fn=eval_input_fn,
            eval_steps=None,
            **experiment_args
        )

    return _experiment_fn

b. Set HParam and RunConfig





In [10]:



    


TRAIN_SIZE = 12000
NUM_EPOCHS = 1000
BATCH_SIZE = 500
NUM_EVAL = 10
CHECKPOINT_STEPS = int((TRAIN_SIZE/BATCH_SIZE) * (NUM_EPOCHS/NUM_EVAL))

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    hidden_units=[8, 4], 
    dropout_prob = 0.0)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.contrib.learn.RunConfig(
    save_checkpoints_steps=CHECKPOINT_STEPS,
    tf_random_seed=19830610,
    model_dir=model_dir
)

print(hparams)
print("Model Directory:", run_config.model_dir)
print("")
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", (TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS)
print("Required Evaluation Steps:", NUM_EVAL) 
print("That is 1 evaluation step after each",NUM_EPOCHS/NUM_EVAL," epochs")
print("Save Checkpoint After",CHECKPOINT_STEPS,"steps")



    


















    






[('batch_size', 500), ('dropout_prob', 0.0), ('hidden_units', [8, 4]), ('num_epochs', 1000)]
Model Directory: trained_models/reg-model-03

Dataset Size: 12000
Batch Size: 500
Steps per Epoch: 24.0
Total Steps: 24000.0
Required Evaluation Steps: 10
That is 1 evaluation step after each 100.0  epochs
Save Checkpoint After 2400 steps

c. Define JSON Serving Function





In [11]:



    


def json_serving_input_fn():
    
    receiver_tensor = {}

    for feature_name in FEATURE_NAMES:
        dtype = tf.float32 if feature_name in NUMERIC_FEATURE_NAMES else tf.string
        receiver_tensor[feature_name] = tf.placeholder(shape=[None], dtype=dtype)

    if PROCESS_FEATURES:
        features = process_features(receiver_tensor)

    return tf.estimator.export.ServingInputReceiver(
        features, receiver_tensor)

d. Run Experiment via learn_runner





In [12]:



    


if not RESUME_TRAINING:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...") 


tf.logging.set_verbosity(tf.logging.INFO)

time_start = datetime.utcnow() 
print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................") 


learn_runner.run(
    experiment_fn=generate_experiment_fn(
        
        export_strategies=[
            make_export_strategy(
            json_serving_input_fn,
            exports_to_keep=1,
            as_text=True)
        ]
    ),
    run_config=run_config,
    schedule="train_and_evaluate",
    hparams=hparams
)

time_end = datetime.utcnow() 
print(".......................................")
print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))



    


















    






Removing previous artifacts...
Experiment started at 10:02:21
.......................................
WARNING:tensorflow:RunConfig.uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.
INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x12228d940>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1
}
, '_tf_random_seed': 19830610, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': 2400, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'trained_models/reg-model-03'}

Estimator Type: <class 'tensorflow.python.estimator.canned.dnn.DNNRegressor'>

WARNING:tensorflow:RunConfig.uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.
WARNING:tensorflow:From /Users/khalidsalama/anaconda/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:267: BaseMonitor.__init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.

* data input_fn:
================
Input file(s): data/train-*.csv
Batch size: 500
Epoch Count: 1000
Mode: train
Shuffle: True
================

INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Saving checkpoints for 1 into trained_models/reg-model-03/model.ckpt.

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:02:26
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-1
INFO:tensorflow:Finished evaluation at 2017-11-16-10:02:26
INFO:tensorflow:Saving dict for global step 1: average_loss = 328.465, global_step = 1, loss = 164232.0
INFO:tensorflow:Validation (step 1): average_loss = 328.465, loss = 164232.0, global_step = 1
INFO:tensorflow:loss = 155660.0, step = 1
INFO:tensorflow:global_step/sec: 40.1198
INFO:tensorflow:loss = 185020.0, step = 101 (0.776 sec)
INFO:tensorflow:global_step/sec: 129.882
INFO:tensorflow:loss = 137635.0, step = 201 (0.771 sec)
INFO:tensorflow:global_step/sec: 125.641
INFO:tensorflow:loss = 171515.0, step = 301 (0.797 sec)
INFO:tensorflow:global_step/sec: 118.015
INFO:tensorflow:loss = 140163.0, step = 401 (0.844 sec)
INFO:tensorflow:global_step/sec: 126.49
INFO:tensorflow:loss = 141613.0, step = 501 (0.795 sec)
INFO:tensorflow:global_step/sec: 116.976
INFO:tensorflow:loss = 136004.0, step = 601 (0.855 sec)
INFO:tensorflow:global_step/sec: 113.562
INFO:tensorflow:loss = 130641.0, step = 701 (0.879 sec)
INFO:tensorflow:global_step/sec: 126.747
INFO:tensorflow:loss = 111425.0, step = 801 (0.788 sec)
INFO:tensorflow:global_step/sec: 131.066
INFO:tensorflow:loss = 113015.0, step = 901 (0.763 sec)
INFO:tensorflow:global_step/sec: 108.453
INFO:tensorflow:loss = 93464.3, step = 1001 (0.922 sec)
INFO:tensorflow:global_step/sec: 139.519
INFO:tensorflow:loss = 87641.5, step = 1101 (0.717 sec)
INFO:tensorflow:global_step/sec: 134.695
INFO:tensorflow:loss = 88079.6, step = 1201 (0.746 sec)
INFO:tensorflow:global_step/sec: 121.82
INFO:tensorflow:loss = 85597.5, step = 1301 (0.817 sec)
INFO:tensorflow:global_step/sec: 121.88
INFO:tensorflow:loss = 63857.3, step = 1401 (0.826 sec)
INFO:tensorflow:global_step/sec: 119.062
INFO:tensorflow:loss = 70258.2, step = 1501 (0.834 sec)
INFO:tensorflow:global_step/sec: 97.4661
INFO:tensorflow:loss = 55564.2, step = 1601 (1.027 sec)
INFO:tensorflow:global_step/sec: 118.683
INFO:tensorflow:loss = 63768.5, step = 1701 (0.842 sec)
INFO:tensorflow:global_step/sec: 104.212
INFO:tensorflow:loss = 54877.1, step = 1801 (0.962 sec)
INFO:tensorflow:global_step/sec: 114.517
INFO:tensorflow:loss = 60213.6, step = 1901 (0.874 sec)
INFO:tensorflow:global_step/sec: 107.671
INFO:tensorflow:loss = 54891.5, step = 2001 (0.925 sec)
INFO:tensorflow:global_step/sec: 136.533
INFO:tensorflow:loss = 47515.1, step = 2101 (0.733 sec)
INFO:tensorflow:global_step/sec: 118.671
INFO:tensorflow:loss = 51576.9, step = 2201 (0.841 sec)
INFO:tensorflow:global_step/sec: 129.906
INFO:tensorflow:loss = 50460.4, step = 2301 (0.770 sec)
INFO:tensorflow:Saving checkpoints for 2401 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 75.2384

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:02:48
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-2401
INFO:tensorflow:Finished evaluation at 2017-11-16-10:02:48
INFO:tensorflow:Saving dict for global step 2401: average_loss = 104.677, global_step = 2401, loss = 52338.7
INFO:tensorflow:Validation (step 2401): average_loss = 104.677, loss = 52338.7, global_step = 2401
INFO:tensorflow:loss = 54241.1, step = 2401 (2.193 sec)
INFO:tensorflow:global_step/sec: 63.4717
INFO:tensorflow:loss = 42089.1, step = 2501 (0.712 sec)
INFO:tensorflow:global_step/sec: 142.283
INFO:tensorflow:loss = 43037.2, step = 2601 (0.702 sec)
INFO:tensorflow:global_step/sec: 137.143
INFO:tensorflow:loss = 46601.3, step = 2701 (0.731 sec)
INFO:tensorflow:global_step/sec: 139.558
INFO:tensorflow:loss = 42919.7, step = 2801 (0.715 sec)
INFO:tensorflow:global_step/sec: 141.816
INFO:tensorflow:loss = 44527.5, step = 2901 (0.705 sec)
INFO:tensorflow:global_step/sec: 140.665
INFO:tensorflow:loss = 47868.0, step = 3001 (0.711 sec)
INFO:tensorflow:global_step/sec: 141.832
INFO:tensorflow:loss = 47298.0, step = 3101 (0.706 sec)
INFO:tensorflow:global_step/sec: 121.252
INFO:tensorflow:loss = 49777.9, step = 3201 (0.825 sec)
INFO:tensorflow:global_step/sec: 90.0606
INFO:tensorflow:loss = 44413.0, step = 3301 (1.122 sec)
INFO:tensorflow:global_step/sec: 96.3119
INFO:tensorflow:loss = 47032.9, step = 3401 (1.026 sec)
INFO:tensorflow:global_step/sec: 120.489
INFO:tensorflow:loss = 36708.7, step = 3501 (0.832 sec)
INFO:tensorflow:global_step/sec: 130.31
INFO:tensorflow:loss = 48602.3, step = 3601 (0.765 sec)
INFO:tensorflow:global_step/sec: 121.533
INFO:tensorflow:loss = 46881.4, step = 3701 (0.822 sec)
INFO:tensorflow:global_step/sec: 134.723
INFO:tensorflow:loss = 44374.4, step = 3801 (0.745 sec)
INFO:tensorflow:global_step/sec: 118.789
INFO:tensorflow:loss = 45259.9, step = 3901 (0.840 sec)
INFO:tensorflow:global_step/sec: 133.68
INFO:tensorflow:loss = 50385.1, step = 4001 (0.752 sec)
INFO:tensorflow:global_step/sec: 123.912
INFO:tensorflow:loss = 46569.6, step = 4101 (0.804 sec)
INFO:tensorflow:global_step/sec: 136.72
INFO:tensorflow:loss = 47248.7, step = 4201 (0.734 sec)
INFO:tensorflow:global_step/sec: 95.0957
INFO:tensorflow:loss = 40043.4, step = 4301 (1.051 sec)
INFO:tensorflow:global_step/sec: 96.5031
INFO:tensorflow:loss = 45197.2, step = 4401 (1.036 sec)
INFO:tensorflow:global_step/sec: 111.985
INFO:tensorflow:loss = 47951.4, step = 4501 (0.892 sec)
INFO:tensorflow:global_step/sec: 136.538
INFO:tensorflow:loss = 44457.1, step = 4601 (0.733 sec)
INFO:tensorflow:global_step/sec: 116.219
INFO:tensorflow:loss = 37337.1, step = 4701 (0.862 sec)
INFO:tensorflow:Saving checkpoints for 4801 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 49.1258

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:03:10
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-4801
INFO:tensorflow:Finished evaluation at 2017-11-16-10:03:10
INFO:tensorflow:Saving dict for global step 4801: average_loss = 96.7373, global_step = 4801, loss = 48368.7
INFO:tensorflow:Validation (step 4801): average_loss = 96.7373, loss = 48368.7, global_step = 4801
INFO:tensorflow:loss = 46016.4, step = 4801 (3.300 sec)
INFO:tensorflow:global_step/sec: 46.0907
INFO:tensorflow:loss = 46275.1, step = 4901 (0.905 sec)
INFO:tensorflow:global_step/sec: 94.0795
INFO:tensorflow:loss = 33856.8, step = 5001 (1.060 sec)
INFO:tensorflow:global_step/sec: 109.878
INFO:tensorflow:loss = 58067.3, step = 5101 (0.911 sec)
INFO:tensorflow:global_step/sec: 124.923
INFO:tensorflow:loss = 36591.6, step = 5201 (0.800 sec)
INFO:tensorflow:global_step/sec: 139.443
INFO:tensorflow:loss = 49899.2, step = 5301 (0.717 sec)
INFO:tensorflow:global_step/sec: 139.537
INFO:tensorflow:loss = 59058.9, step = 5401 (0.716 sec)
INFO:tensorflow:global_step/sec: 133.873
INFO:tensorflow:loss = 39515.4, step = 5501 (0.747 sec)
INFO:tensorflow:global_step/sec: 134.871
INFO:tensorflow:loss = 37766.9, step = 5601 (0.741 sec)
INFO:tensorflow:global_step/sec: 121.034
INFO:tensorflow:loss = 45288.4, step = 5701 (0.826 sec)
INFO:tensorflow:global_step/sec: 116.242
INFO:tensorflow:loss = 41809.8, step = 5801 (0.861 sec)
INFO:tensorflow:global_step/sec: 131.099
INFO:tensorflow:loss = 42418.6, step = 5901 (0.763 sec)
INFO:tensorflow:global_step/sec: 104.885
INFO:tensorflow:loss = 49231.9, step = 6001 (0.955 sec)
INFO:tensorflow:global_step/sec: 133.241
INFO:tensorflow:loss = 37564.2, step = 6101 (0.749 sec)
INFO:tensorflow:global_step/sec: 119.417
INFO:tensorflow:loss = 42841.4, step = 6201 (0.838 sec)
INFO:tensorflow:global_step/sec: 111.944
INFO:tensorflow:loss = 47508.5, step = 6301 (0.892 sec)
INFO:tensorflow:global_step/sec: 124.069
INFO:tensorflow:loss = 44208.4, step = 6401 (0.807 sec)
INFO:tensorflow:global_step/sec: 132.681
INFO:tensorflow:loss = 40910.1, step = 6501 (0.754 sec)
INFO:tensorflow:global_step/sec: 141.678
INFO:tensorflow:loss = 50287.4, step = 6601 (0.706 sec)
INFO:tensorflow:global_step/sec: 130.994
INFO:tensorflow:loss = 39495.4, step = 6701 (0.766 sec)
INFO:tensorflow:global_step/sec: 122.829
INFO:tensorflow:loss = 43742.4, step = 6801 (0.811 sec)
INFO:tensorflow:global_step/sec: 137.6
INFO:tensorflow:loss = 44626.8, step = 6901 (0.728 sec)
INFO:tensorflow:global_step/sec: 135.7
INFO:tensorflow:loss = 40497.6, step = 7001 (0.735 sec)
INFO:tensorflow:global_step/sec: 141.821
INFO:tensorflow:loss = 37368.0, step = 7101 (0.707 sec)
INFO:tensorflow:Saving checkpoints for 7201 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 74.2526

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:03:31
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-7201
INFO:tensorflow:Finished evaluation at 2017-11-16-10:03:31
INFO:tensorflow:Saving dict for global step 7201: average_loss = 95.7069, global_step = 7201, loss = 47853.4
INFO:tensorflow:Validation (step 7201): average_loss = 95.7069, loss = 47853.4, global_step = 7201
INFO:tensorflow:loss = 51782.6, step = 7201 (2.507 sec)
INFO:tensorflow:global_step/sec: 48.8914
INFO:tensorflow:loss = 42918.2, step = 7301 (0.885 sec)
INFO:tensorflow:global_step/sec: 102.411
INFO:tensorflow:loss = 40595.4, step = 7401 (0.976 sec)
INFO:tensorflow:global_step/sec: 130.36
INFO:tensorflow:loss = 29395.8, step = 7501 (0.767 sec)
INFO:tensorflow:global_step/sec: 94.1363
INFO:tensorflow:loss = 51844.5, step = 7601 (1.066 sec)
INFO:tensorflow:global_step/sec: 79.7989
INFO:tensorflow:loss = 43390.5, step = 7701 (1.253 sec)
INFO:tensorflow:global_step/sec: 117.166
INFO:tensorflow:loss = 47932.0, step = 7801 (0.854 sec)
INFO:tensorflow:global_step/sec: 121.538
INFO:tensorflow:loss = 50953.5, step = 7901 (0.819 sec)
INFO:tensorflow:global_step/sec: 123.403
INFO:tensorflow:loss = 45364.8, step = 8001 (0.811 sec)
INFO:tensorflow:global_step/sec: 134.762
INFO:tensorflow:loss = 37158.0, step = 8101 (0.742 sec)
INFO:tensorflow:global_step/sec: 139.909
INFO:tensorflow:loss = 51231.4, step = 8201 (0.716 sec)
INFO:tensorflow:global_step/sec: 134.113
INFO:tensorflow:loss = 45554.2, step = 8301 (0.744 sec)
INFO:tensorflow:global_step/sec: 139.928
INFO:tensorflow:loss = 38198.0, step = 8401 (0.715 sec)
INFO:tensorflow:global_step/sec: 130.906
INFO:tensorflow:loss = 37683.4, step = 8501 (0.764 sec)
INFO:tensorflow:global_step/sec: 133.234
INFO:tensorflow:loss = 48373.8, step = 8601 (0.751 sec)
INFO:tensorflow:global_step/sec: 132.825
INFO:tensorflow:loss = 39778.3, step = 8701 (0.753 sec)
INFO:tensorflow:global_step/sec: 132.1
INFO:tensorflow:loss = 46058.1, step = 8801 (0.759 sec)
INFO:tensorflow:global_step/sec: 137.113
INFO:tensorflow:loss = 46736.0, step = 8901 (0.728 sec)
INFO:tensorflow:global_step/sec: 139.206
INFO:tensorflow:loss = 40734.2, step = 9001 (0.717 sec)
INFO:tensorflow:global_step/sec: 134.553
INFO:tensorflow:loss = 43216.0, step = 9101 (0.744 sec)
INFO:tensorflow:global_step/sec: 133.409
INFO:tensorflow:loss = 51448.4, step = 9201 (0.750 sec)
INFO:tensorflow:global_step/sec: 133.238
INFO:tensorflow:loss = 46285.5, step = 9301 (0.753 sec)
INFO:tensorflow:global_step/sec: 117.503
INFO:tensorflow:loss = 45954.6, step = 9401 (0.849 sec)
INFO:tensorflow:global_step/sec: 111.311
INFO:tensorflow:loss = 48068.8, step = 9501 (0.901 sec)
INFO:tensorflow:Saving checkpoints for 9601 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 60.0009

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:03:52
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-9601
INFO:tensorflow:Finished evaluation at 2017-11-16-10:03:53
INFO:tensorflow:Saving dict for global step 9601: average_loss = 95.0352, global_step = 9601, loss = 47517.6
INFO:tensorflow:Validation (step 9601): average_loss = 95.0352, loss = 47517.6, global_step = 9601
INFO:tensorflow:loss = 53438.2, step = 9601 (2.797 sec)
INFO:tensorflow:global_step/sec: 50.6965
INFO:tensorflow:loss = 53259.5, step = 9701 (0.840 sec)
INFO:tensorflow:global_step/sec: 118.401
INFO:tensorflow:loss = 34251.2, step = 9801 (0.844 sec)
INFO:tensorflow:global_step/sec: 123.185
INFO:tensorflow:loss = 42877.2, step = 9901 (0.810 sec)
INFO:tensorflow:global_step/sec: 123.078
INFO:tensorflow:loss = 49685.0, step = 10001 (0.814 sec)
INFO:tensorflow:global_step/sec: 110.534
INFO:tensorflow:loss = 47262.9, step = 10101 (0.905 sec)
INFO:tensorflow:global_step/sec: 106.524
INFO:tensorflow:loss = 49317.5, step = 10201 (0.938 sec)
INFO:tensorflow:global_step/sec: 111.962
INFO:tensorflow:loss = 41488.7, step = 10301 (0.893 sec)
INFO:tensorflow:global_step/sec: 134.916
INFO:tensorflow:loss = 40965.7, step = 10401 (0.740 sec)
INFO:tensorflow:global_step/sec: 137.907
INFO:tensorflow:loss = 34246.2, step = 10501 (0.726 sec)
INFO:tensorflow:global_step/sec: 138.882
INFO:tensorflow:loss = 41920.2, step = 10601 (0.719 sec)
INFO:tensorflow:global_step/sec: 138.611
INFO:tensorflow:loss = 40829.2, step = 10701 (0.721 sec)
INFO:tensorflow:global_step/sec: 140.418
INFO:tensorflow:loss = 53730.8, step = 10801 (0.712 sec)
INFO:tensorflow:global_step/sec: 136.933
INFO:tensorflow:loss = 39976.0, step = 10901 (0.731 sec)
INFO:tensorflow:global_step/sec: 134.049
INFO:tensorflow:loss = 42867.9, step = 11001 (0.746 sec)
INFO:tensorflow:global_step/sec: 130.021
INFO:tensorflow:loss = 44854.7, step = 11101 (0.769 sec)
INFO:tensorflow:global_step/sec: 108.442
INFO:tensorflow:loss = 43357.3, step = 11201 (0.928 sec)
INFO:tensorflow:global_step/sec: 109.895
INFO:tensorflow:loss = 44547.3, step = 11301 (0.905 sec)
INFO:tensorflow:global_step/sec: 117.914
INFO:tensorflow:loss = 41676.9, step = 11401 (0.848 sec)
INFO:tensorflow:global_step/sec: 147.302
INFO:tensorflow:loss = 30990.5, step = 11501 (0.677 sec)
INFO:tensorflow:global_step/sec: 145.749
INFO:tensorflow:loss = 50680.9, step = 11601 (0.688 sec)
INFO:tensorflow:global_step/sec: 132.172
INFO:tensorflow:loss = 41594.9, step = 11701 (0.758 sec)
INFO:tensorflow:global_step/sec: 136.047
INFO:tensorflow:loss = 42287.2, step = 11801 (0.733 sec)
INFO:tensorflow:global_step/sec: 147.073
INFO:tensorflow:loss = 49489.5, step = 11901 (0.679 sec)
INFO:tensorflow:Saving checkpoints for 12001 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 80.9467

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:04:13
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-12001
INFO:tensorflow:Finished evaluation at 2017-11-16-10:04:13
INFO:tensorflow:Saving dict for global step 12001: average_loss = 94.4759, global_step = 12001, loss = 47237.9
INFO:tensorflow:Validation (step 12001): average_loss = 94.4759, loss = 47237.9, global_step = 12001
INFO:tensorflow:loss = 48240.1, step = 12001 (2.193 sec)
INFO:tensorflow:global_step/sec: 59.5909
INFO:tensorflow:loss = 33612.9, step = 12101 (0.720 sec)
INFO:tensorflow:global_step/sec: 137.643
INFO:tensorflow:loss = 38861.4, step = 12201 (0.728 sec)
INFO:tensorflow:global_step/sec: 137.798
INFO:tensorflow:loss = 42655.6, step = 12301 (0.725 sec)
INFO:tensorflow:global_step/sec: 136.279
INFO:tensorflow:loss = 35384.6, step = 12401 (0.733 sec)
INFO:tensorflow:global_step/sec: 134.919
INFO:tensorflow:loss = 43430.5, step = 12501 (0.742 sec)
INFO:tensorflow:global_step/sec: 140.924
INFO:tensorflow:loss = 48275.4, step = 12601 (0.709 sec)
INFO:tensorflow:global_step/sec: 145.792
INFO:tensorflow:loss = 37730.8, step = 12701 (0.685 sec)
INFO:tensorflow:global_step/sec: 145.99
INFO:tensorflow:loss = 48645.6, step = 12801 (0.685 sec)
INFO:tensorflow:global_step/sec: 143.333
INFO:tensorflow:loss = 30703.5, step = 12901 (0.698 sec)
INFO:tensorflow:global_step/sec: 142.704
INFO:tensorflow:loss = 43611.7, step = 13001 (0.700 sec)
INFO:tensorflow:global_step/sec: 142.848
INFO:tensorflow:loss = 38099.9, step = 13101 (0.701 sec)
INFO:tensorflow:global_step/sec: 135.326
INFO:tensorflow:loss = 45194.5, step = 13201 (0.739 sec)
INFO:tensorflow:global_step/sec: 135.168
INFO:tensorflow:loss = 38871.8, step = 13301 (0.740 sec)
INFO:tensorflow:global_step/sec: 129.326
INFO:tensorflow:loss = 36956.7, step = 13401 (0.773 sec)
INFO:tensorflow:global_step/sec: 125.692
INFO:tensorflow:loss = 39388.8, step = 13501 (0.796 sec)
INFO:tensorflow:global_step/sec: 126.122
INFO:tensorflow:loss = 46446.1, step = 13601 (0.791 sec)
INFO:tensorflow:global_step/sec: 136.738
INFO:tensorflow:loss = 44792.6, step = 13701 (0.730 sec)
INFO:tensorflow:global_step/sec: 144.558
INFO:tensorflow:loss = 55869.1, step = 13801 (0.693 sec)
INFO:tensorflow:global_step/sec: 122.661
INFO:tensorflow:loss = 42912.5, step = 13901 (0.818 sec)
INFO:tensorflow:global_step/sec: 124.063
INFO:tensorflow:loss = 44276.9, step = 14001 (0.803 sec)
INFO:tensorflow:global_step/sec: 140.367
INFO:tensorflow:loss = 32231.6, step = 14101 (0.713 sec)
INFO:tensorflow:global_step/sec: 128.227
INFO:tensorflow:loss = 33367.5, step = 14201 (0.779 sec)
INFO:tensorflow:global_step/sec: 121.112
INFO:tensorflow:loss = 43804.7, step = 14301 (0.827 sec)
INFO:tensorflow:Saving checkpoints for 14401 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 53.1746

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:04:33
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-14401
INFO:tensorflow:Finished evaluation at 2017-11-16-10:04:33
INFO:tensorflow:Saving dict for global step 14401: average_loss = 93.9907, global_step = 14401, loss = 46995.4
INFO:tensorflow:Validation (step 14401): average_loss = 93.9907, loss = 46995.4, global_step = 14401
INFO:tensorflow:loss = 57803.0, step = 14401 (3.205 sec)
INFO:tensorflow:global_step/sec: 46.7932
INFO:tensorflow:loss = 46377.5, step = 14501 (0.813 sec)
INFO:tensorflow:global_step/sec: 134.385
INFO:tensorflow:loss = 34797.7, step = 14601 (0.744 sec)
INFO:tensorflow:global_step/sec: 112.674
INFO:tensorflow:loss = 44715.1, step = 14701 (0.888 sec)
INFO:tensorflow:global_step/sec: 130.88
INFO:tensorflow:loss = 39205.6, step = 14801 (0.764 sec)
INFO:tensorflow:global_step/sec: 135.123
INFO:tensorflow:loss = 40149.7, step = 14901 (0.740 sec)
INFO:tensorflow:global_step/sec: 133.989
INFO:tensorflow:loss = 43184.4, step = 15001 (0.747 sec)
INFO:tensorflow:global_step/sec: 134.434
INFO:tensorflow:loss = 38481.5, step = 15101 (0.744 sec)
INFO:tensorflow:global_step/sec: 104.198
INFO:tensorflow:loss = 40600.9, step = 15201 (0.959 sec)
INFO:tensorflow:global_step/sec: 137.205
INFO:tensorflow:loss = 49492.6, step = 15301 (0.729 sec)
INFO:tensorflow:global_step/sec: 134.984
INFO:tensorflow:loss = 29930.4, step = 15401 (0.741 sec)
INFO:tensorflow:global_step/sec: 137.089
INFO:tensorflow:loss = 46129.1, step = 15501 (0.730 sec)
INFO:tensorflow:global_step/sec: 132.123
INFO:tensorflow:loss = 47175.0, step = 15601 (0.758 sec)
INFO:tensorflow:global_step/sec: 103.908
INFO:tensorflow:loss = 39116.7, step = 15701 (0.963 sec)
INFO:tensorflow:global_step/sec: 95.1968
INFO:tensorflow:loss = 35493.4, step = 15801 (1.057 sec)
INFO:tensorflow:global_step/sec: 124.185
INFO:tensorflow:loss = 41123.8, step = 15901 (0.797 sec)
INFO:tensorflow:global_step/sec: 119.053
INFO:tensorflow:loss = 28964.1, step = 16001 (0.843 sec)
INFO:tensorflow:global_step/sec: 130.346
INFO:tensorflow:loss = 42644.8, step = 16101 (0.766 sec)
INFO:tensorflow:global_step/sec: 126.435
INFO:tensorflow:loss = 41032.1, step = 16201 (0.791 sec)
INFO:tensorflow:global_step/sec: 134.572
INFO:tensorflow:loss = 38422.3, step = 16301 (0.742 sec)
INFO:tensorflow:global_step/sec: 139.46
INFO:tensorflow:loss = 40377.9, step = 16401 (0.718 sec)
INFO:tensorflow:global_step/sec: 138.869
INFO:tensorflow:loss = 45031.1, step = 16501 (0.720 sec)
INFO:tensorflow:global_step/sec: 140.669
INFO:tensorflow:loss = 44225.1, step = 16601 (0.711 sec)
INFO:tensorflow:global_step/sec: 137.776
INFO:tensorflow:loss = 44445.1, step = 16701 (0.727 sec)
INFO:tensorflow:Saving checkpoints for 16801 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 71.631

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:04:54
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-16801
INFO:tensorflow:Finished evaluation at 2017-11-16-10:04:54
INFO:tensorflow:Saving dict for global step 16801: average_loss = 93.6571, global_step = 16801, loss = 46828.6
INFO:tensorflow:Validation (step 16801): average_loss = 93.6571, loss = 46828.6, global_step = 16801
INFO:tensorflow:loss = 43306.4, step = 16801 (2.605 sec)
INFO:tensorflow:global_step/sec: 46.2648
INFO:tensorflow:loss = 54241.1, step = 16901 (0.952 sec)
INFO:tensorflow:global_step/sec: 140.135
INFO:tensorflow:loss = 34022.9, step = 17001 (0.714 sec)
INFO:tensorflow:global_step/sec: 130.628
INFO:tensorflow:loss = 45510.8, step = 17101 (0.766 sec)
INFO:tensorflow:global_step/sec: 140.74
INFO:tensorflow:loss = 43210.0, step = 17201 (0.710 sec)
INFO:tensorflow:global_step/sec: 138.249
INFO:tensorflow:loss = 50877.1, step = 17301 (0.723 sec)
INFO:tensorflow:global_step/sec: 135.93
INFO:tensorflow:loss = 38702.2, step = 17401 (0.735 sec)
INFO:tensorflow:global_step/sec: 137.277
INFO:tensorflow:loss = 34215.1, step = 17501 (0.729 sec)
INFO:tensorflow:global_step/sec: 134.199
INFO:tensorflow:loss = 39389.1, step = 17601 (0.746 sec)
INFO:tensorflow:global_step/sec: 130.333
INFO:tensorflow:loss = 43825.1, step = 17701 (0.767 sec)
INFO:tensorflow:global_step/sec: 131.614
INFO:tensorflow:loss = 43972.8, step = 17801 (0.760 sec)
INFO:tensorflow:global_step/sec: 134.144
INFO:tensorflow:loss = 48032.7, step = 17901 (0.745 sec)
INFO:tensorflow:global_step/sec: 139.993
INFO:tensorflow:loss = 43332.0, step = 18001 (0.715 sec)
INFO:tensorflow:global_step/sec: 142.514
INFO:tensorflow:loss = 39407.1, step = 18101 (0.702 sec)
INFO:tensorflow:global_step/sec: 138.503
INFO:tensorflow:loss = 46066.7, step = 18201 (0.722 sec)
INFO:tensorflow:global_step/sec: 126.932
INFO:tensorflow:loss = 43000.3, step = 18301 (0.788 sec)
INFO:tensorflow:global_step/sec: 136.917
INFO:tensorflow:loss = 43819.3, step = 18401 (0.731 sec)
INFO:tensorflow:global_step/sec: 117.591
INFO:tensorflow:loss = 43357.4, step = 18501 (0.850 sec)
INFO:tensorflow:global_step/sec: 107.758
INFO:tensorflow:loss = 50256.3, step = 18601 (0.930 sec)
INFO:tensorflow:global_step/sec: 104.861
INFO:tensorflow:loss = 42203.0, step = 18701 (0.956 sec)
INFO:tensorflow:global_step/sec: 111.49
INFO:tensorflow:loss = 39352.0, step = 18801 (0.894 sec)
INFO:tensorflow:global_step/sec: 114.315
INFO:tensorflow:loss = 44110.5, step = 18901 (0.874 sec)
INFO:tensorflow:global_step/sec: 119.923
INFO:tensorflow:loss = 52128.4, step = 19001 (0.834 sec)
INFO:tensorflow:global_step/sec: 124.208
INFO:tensorflow:loss = 44822.1, step = 19101 (0.804 sec)
INFO:tensorflow:Saving checkpoints for 19201 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 49.8886

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:05:15
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-19201
INFO:tensorflow:Finished evaluation at 2017-11-16-10:05:16
INFO:tensorflow:Saving dict for global step 19201: average_loss = 93.4035, global_step = 19201, loss = 46701.7
INFO:tensorflow:Validation (step 19201): average_loss = 93.4035, loss = 46701.7, global_step = 19201
INFO:tensorflow:loss = 41447.1, step = 19201 (3.591 sec)
INFO:tensorflow:global_step/sec: 38.6598
INFO:tensorflow:loss = 39285.7, step = 19301 (1.002 sec)
INFO:tensorflow:global_step/sec: 99.165
INFO:tensorflow:loss = 47136.1, step = 19401 (1.010 sec)
INFO:tensorflow:global_step/sec: 93.02
INFO:tensorflow:loss = 48500.1, step = 19501 (1.072 sec)
INFO:tensorflow:global_step/sec: 104.564
INFO:tensorflow:loss = 41969.5, step = 19601 (0.955 sec)
INFO:tensorflow:global_step/sec: 138.815
INFO:tensorflow:loss = 45881.1, step = 19701 (0.722 sec)
INFO:tensorflow:global_step/sec: 104.89
INFO:tensorflow:loss = 38483.6, step = 19801 (0.957 sec)
INFO:tensorflow:global_step/sec: 97.0502
INFO:tensorflow:loss = 38991.4, step = 19901 (1.026 sec)
INFO:tensorflow:global_step/sec: 117.119
INFO:tensorflow:loss = 33769.5, step = 20001 (0.852 sec)
INFO:tensorflow:global_step/sec: 114.517
INFO:tensorflow:loss = 37037.8, step = 20101 (0.874 sec)
INFO:tensorflow:global_step/sec: 101.101
INFO:tensorflow:loss = 44271.2, step = 20201 (0.996 sec)
INFO:tensorflow:global_step/sec: 121.698
INFO:tensorflow:loss = 47297.9, step = 20301 (0.814 sec)
INFO:tensorflow:global_step/sec: 131.795
INFO:tensorflow:loss = 49067.2, step = 20401 (0.759 sec)
INFO:tensorflow:global_step/sec: 130.245
INFO:tensorflow:loss = 29920.0, step = 20501 (0.768 sec)
INFO:tensorflow:global_step/sec: 117.683
INFO:tensorflow:loss = 41268.2, step = 20601 (0.853 sec)
INFO:tensorflow:global_step/sec: 117.073
INFO:tensorflow:loss = 48263.5, step = 20701 (0.851 sec)
INFO:tensorflow:global_step/sec: 115.17
INFO:tensorflow:loss = 43600.9, step = 20801 (0.867 sec)
INFO:tensorflow:global_step/sec: 132.257
INFO:tensorflow:loss = 53666.0, step = 20901 (0.758 sec)
INFO:tensorflow:global_step/sec: 115.403
INFO:tensorflow:loss = 43610.4, step = 21001 (0.867 sec)
INFO:tensorflow:global_step/sec: 113.231
INFO:tensorflow:loss = 39353.9, step = 21101 (0.885 sec)
INFO:tensorflow:global_step/sec: 117.822
INFO:tensorflow:loss = 41081.1, step = 21201 (0.846 sec)
INFO:tensorflow:global_step/sec: 108.78
INFO:tensorflow:loss = 43726.5, step = 21301 (0.920 sec)
INFO:tensorflow:global_step/sec: 101.509
INFO:tensorflow:loss = 46147.2, step = 21401 (0.988 sec)
INFO:tensorflow:global_step/sec: 106.962
INFO:tensorflow:loss = 42668.0, step = 21501 (0.933 sec)
INFO:tensorflow:Saving checkpoints for 21601 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 68.8721

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:05:38
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-21601
INFO:tensorflow:Finished evaluation at 2017-11-16-10:05:39
INFO:tensorflow:Saving dict for global step 21601: average_loss = 93.2296, global_step = 21601, loss = 46614.8
INFO:tensorflow:Validation (step 21601): average_loss = 93.2296, loss = 46614.8, global_step = 21601
INFO:tensorflow:loss = 39012.0, step = 21601 (2.402 sec)
INFO:tensorflow:global_step/sec: 55.8269
INFO:tensorflow:loss = 42405.3, step = 21701 (0.839 sec)
INFO:tensorflow:global_step/sec: 119.44
INFO:tensorflow:loss = 47157.8, step = 21801 (0.837 sec)
INFO:tensorflow:global_step/sec: 110.084
INFO:tensorflow:loss = 40285.0, step = 21901 (0.910 sec)
INFO:tensorflow:global_step/sec: 114.772
INFO:tensorflow:loss = 51357.3, step = 22001 (0.873 sec)
INFO:tensorflow:global_step/sec: 129.632
INFO:tensorflow:loss = 43420.6, step = 22101 (0.770 sec)
INFO:tensorflow:global_step/sec: 137.865
INFO:tensorflow:loss = 50706.4, step = 22201 (0.725 sec)
INFO:tensorflow:global_step/sec: 116.058
INFO:tensorflow:loss = 47902.4, step = 22301 (0.863 sec)
INFO:tensorflow:global_step/sec: 120.49
INFO:tensorflow:loss = 34552.7, step = 22401 (0.829 sec)
INFO:tensorflow:global_step/sec: 116.685
INFO:tensorflow:loss = 40107.3, step = 22501 (0.856 sec)
INFO:tensorflow:global_step/sec: 127.137
INFO:tensorflow:loss = 43026.8, step = 22601 (0.786 sec)
INFO:tensorflow:global_step/sec: 133.658
INFO:tensorflow:loss = 38352.9, step = 22701 (0.751 sec)
INFO:tensorflow:global_step/sec: 143.827
INFO:tensorflow:loss = 47005.3, step = 22801 (0.693 sec)
INFO:tensorflow:global_step/sec: 126.76
INFO:tensorflow:loss = 34692.8, step = 22901 (0.788 sec)
INFO:tensorflow:global_step/sec: 123.352
INFO:tensorflow:loss = 45891.2, step = 23001 (0.811 sec)
INFO:tensorflow:global_step/sec: 123.05
INFO:tensorflow:loss = 44916.1, step = 23101 (0.813 sec)
INFO:tensorflow:global_step/sec: 131.7
INFO:tensorflow:loss = 42158.5, step = 23201 (0.758 sec)
INFO:tensorflow:global_step/sec: 149.588
INFO:tensorflow:loss = 34986.8, step = 23301 (0.668 sec)
INFO:tensorflow:global_step/sec: 115.691
INFO:tensorflow:loss = 49937.9, step = 23401 (0.865 sec)
INFO:tensorflow:global_step/sec: 145.735
INFO:tensorflow:loss = 51601.8, step = 23501 (0.685 sec)
INFO:tensorflow:global_step/sec: 96.7756
INFO:tensorflow:loss = 44515.1, step = 23601 (1.039 sec)
INFO:tensorflow:global_step/sec: 121.551
INFO:tensorflow:loss = 43017.3, step = 23701 (0.817 sec)
INFO:tensorflow:global_step/sec: 114.851
INFO:tensorflow:loss = 34417.2, step = 23801 (0.876 sec)
INFO:tensorflow:global_step/sec: 109.275
INFO:tensorflow:loss = 45028.1, step = 23901 (0.911 sec)
INFO:tensorflow:Saving checkpoints for 24000 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:Loss for final step: 41759.8.

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:05:59
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-24000
INFO:tensorflow:Finished evaluation at 2017-11-16-10:06:00
INFO:tensorflow:Saving dict for global step 24000: average_loss = 93.1534, global_step = 24000, loss = 46576.7
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-24000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: b"trained_models/reg-model-03/export/Servo/temp-b'1510826760'/saved_model.pbtxt"
.......................................
Experiment finished at 10:06:01

Experiment elapsed time: 219.736155 seconds

6. Evaluate the Model





In [13]:



    


TRAIN_SIZE = 12000
VALID_SIZE = 3000
TEST_SIZE = 5000

train_input_fn = lambda: csv_input_fn(files_name_pattern= TRAIN_DATA_FILES_PATTERN, 
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= TRAIN_SIZE)

valid_input_fn = lambda: csv_input_fn(files_name_pattern= VALID_DATA_FILES_PATTERN, 
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= VALID_SIZE)

test_input_fn = lambda: csv_input_fn(files_name_pattern= TEST_DATA_FILES_PATTERN, 
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= TEST_SIZE)

estimator = create_estimator(run_config, hparams)

train_results = estimator.evaluate(input_fn=train_input_fn, steps=1)
train_rmse = round(math.sqrt(train_results["average_loss"]),5)
print()
print("############################################################################################")
print("# Train RMSE: {} - {}".format(train_rmse, train_results))
print("############################################################################################")

valid_results = estimator.evaluate(input_fn=valid_input_fn, steps=1)
valid_rmse = round(math.sqrt(valid_results["average_loss"]),5)
print()
print("############################################################################################")
print("# Valid RMSE: {} - {}".format(valid_rmse,valid_results))
print("############################################################################################")

test_results = estimator.evaluate(input_fn=test_input_fn, steps=1)
test_rmse = round(math.sqrt(test_results["average_loss"]),5)
print()
print("############################################################################################")
print("# Test RMSE: {} - {}".format(test_rmse, test_results))
print("############################################################################################")



    


















    






INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x12228d940>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1
}
, '_tf_random_seed': 19830610, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': 2400, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'trained_models/reg-model-03'}

Estimator Type: <class 'tensorflow.python.estimator.canned.dnn.DNNRegressor'>


* data input_fn:
================
Input file(s): data/train-*.csv
Batch size: 12000
Epoch Count: None
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:06:01
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-24000
INFO:tensorflow:Evaluation [1/1]
INFO:tensorflow:Finished evaluation at 2017-11-16-10:06:02
INFO:tensorflow:Saving dict for global step 24000: average_loss = 84.9214, global_step = 24000, loss = 1.01906e+06

############################################################################################
# Train RMSE: 9.21528 - {'average_loss': 84.921417, 'loss': 1019057.0, 'global_step': 24000}
############################################################################################

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 3000
Epoch Count: None
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:06:02
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-24000
INFO:tensorflow:Evaluation [1/1]
INFO:tensorflow:Finished evaluation at 2017-11-16-10:06:03
INFO:tensorflow:Saving dict for global step 24000: average_loss = 93.1534, global_step = 24000, loss = 279460.0

############################################################################################
# Valid RMSE: 9.6516 - {'average_loss': 93.153397, 'loss': 279460.19, 'global_step': 24000}
############################################################################################

* data input_fn:
================
Input file(s): data/test-*.csv
Batch size: 5000
Epoch Count: None
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-16-10:06:03
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-24000
INFO:tensorflow:Evaluation [1/1]
INFO:tensorflow:Finished evaluation at 2017-11-16-10:06:04
INFO:tensorflow:Saving dict for global step 24000: average_loss = 93.2213, global_step = 24000, loss = 466107.0

############################################################################################
# Test RMSE: 9.65512 - {'average_loss': 93.221344, 'loss': 466106.72, 'global_step': 24000}
############################################################################################

7. Prediction





In [14]:



    


import itertools

predict_input_fn = lambda: csv_input_fn(files_name_pattern=TEST_DATA_FILES_PATTERN, 
                                      mode= tf.estimator.ModeKeys.PREDICT,
                                      batch_size= 5)

predictions = estimator.predict(input_fn=predict_input_fn)
values = list(map(lambda item: item["predictions"][0],list(itertools.islice(predictions, 5))))
print()
print("Predicted Values: {}".format(values))



    


















    






* data input_fn:
================
Input file(s): data/test-*.csv
Batch size: 5
Epoch Count: None
Mode: infer
Shuffle: False
================

WARNING:tensorflow:Input graph does not contain a QueueRunner. That means predict yields forever. This is probably a mistake.
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-24000

Predicted Values: [51.019321, -5.8079214, 19.57333, 2.9324729, 1.5238302]

Serving via the Saved Model





In [15]:



    


import os

export_dir = model_dir +"/export/Servo/"

saved_model_dir = export_dir + "/" + os.listdir(path=export_dir)[-1] 

print(saved_model_dir)
print("")


predictor_fn = tf.contrib.predictor.from_saved_model(
    export_dir = saved_model_dir,
    signature_def_key="predict"
)

output = predictor_fn(
    {
        'x': [0.5, -1],
        'y': [1, 0.5],
        'alpha': ['ax01', 'ax01'],
        'beta': ['bx02', 'bx01']
        
    }
)
print(output)



    


















    






trained_models/reg-model-03/export/Servo//1510826760

INFO:tensorflow:Restoring parameters from b'trained_models/reg-model-03/export/Servo//1510826760/variables/variables'
{'predictions': array([[ 66.11113739],
       [-11.73743343]], dtype=float32)}

What can we improve?

  • Use .tfrecords files instead of CSV - TFRecord files are optimised for tensorflow.
  • Build a Custom Estimator - Custom Estimator APIs give you the flexibility to build custom models in a simple and standard way




In [ ]:

Content source: GoogleCloudPlatform/tf-estimator-tutorials

Similar notebooks:

notebook.community | gallery | about